In a practical scenario, multi-view video systems involving a large number of cameras might be built using heterogeneous cameras, or cameras that have not been perfectly calibrated. This leads to differences in luminance and chrominance when the same parts of a scene are viewed with different cameras. Moreover, camera distance and positioning also affects illumination, in the sense that the same surface may reflect the light differently when perceived from different angles. Under these scenarios, luminance and chrominance differences will decrease the efficiency of cross-view prediction.
Illumination compensation (IC) is a coding technique to compensate for illumination/color differences in coding multi-view materials. Compared to the International Organization for Standardization/International Electrotechnical Commission (ISO/IEC) Moving Picture Experts Group-4 (MPEG-4) Part 10 Advanced Video Coding (AVC) standard/International Telecommunication Union, Telecommunication Sector (ITU-T) H.264 recommendation (hereinafter the “MPEG-4 AVC standard”), illumination compensation employs predictive coding for the direct current (DC) component of Inter prediction residues. The predictor for illumination change is formed from neighboring blocks to explore the strong spatial correlation of illumination differences.
The local ICA (Illumination change-adaptive) MC (motion compensation) method aims to compensate local illumination changes between pictures in multi-view sequences.
Illumination change-adaptive motion compensation is enabled for the following macroblock modes in the multi-view video coding extension of the MPEG-4 AVC Standard: Inter 16×16 mode; Direct 16×16 mode (including B_Skip); and P_Skip mode.
If mb_ic_flag is equal to zero for the current macroblock (MB), then illumination change-adaptive motion compensation is not performed, i.e., the conventional decoding process is performed. Otherwise, the differential pulse code modulation (DPCM) value of the differential value of illumination compensation (DVIC), namely dpcm_of_dvic, is used, and the proposed illumination change-adaptive motion compensation is performed as follows:DVIC=dpcm_of_dvic+predDVIC f′(i,j)={MR—R″(x′,y′,i,j)+r(i+x′,j+y′)}+DVICwhere f′(i,j) represents the reconstructed partition before deblocking filtering, r(i,j) represents the reference frame, and MR_R″(i,j) represents the reconstructed residual signal. The predictor predDVIC is obtained from the neighboring blocks by using the prediction process which is defined in the MPEG-4 AVC Standard. When illumination compensation is used in bi-predictive coding, there is only one dpcm_of_dvic signaled, and the reconstructed differential value of illumination compensation value is added to the prediction signal. This is equivalent to using two identical values for the differential value of illumination compensation for forward and backward prediction signals.
For illumination change-adaptive motion compensation, illumination compensation can be used for both Skip and Direct modes.
For P skip mode, both mb_ic_flag and dpcm_of_dvic are derived from neighboring Macroblocks. The syntax mb_ic_flag is set to one if illumination compensation is enabled for either the upper macroblock or the left macroblock. The average value of the differential value of illumination compensation from the upper and left macroblocks is used as an illumination compensation parameter of the current macroblock if both neighboring macroblocks use illumination compensation. If only one neighboring macroblock uses illumination compensation, the differential value of illumination compensation of that macroblock is used for the current macroblock.
For Direct—16×16 and B_Skip mode for B_SLICE, both mb_ic flag and dpcm_of_dvic are present in the bitstream.
The prediction process of differential value of illumination compensation will now be described.
Differential value of illumination compensation refers to the local illumination change in each macroblock. The differential pulse code modulation of differential value of illumination compensation, namely dpcm_of_dvic, is written to the bitstream using the predictor (predDVIC) obtained from the differential value of illumination compensation of the neighboring macroblocks as shown in FIG. 3. Turning to FIG. 3, an exemplary layout of block showing the neighboring blocks used for the prediction of the current differential value of illumination compensation (DVIC) is indicated generally by the reference numeral 300. In FIG. 3, the current macroblock to be encoded is indicated by the reference numeral 310, and the neighboring macroblocks used to predict the current macroblock 310 are indicated by the reference numeral 305.
The predDVIC is obtained by performing the following steps:
(1) If the above macroblock “A” of the current macroblock was encoded using macroblock-based illumination change-adaptive motion compensation and its reference index is equal to a reference index of the current macroblock, then predDVIC is set to the differential value of illumination compensation of macroblock “A” and the process is finalized. Otherwise, go to next step.
(2) If the left macroblock “B” of the current macroblock was encoded using macroblock-based illumination change-adaptive motion compensation and its reference index is equal to a reference index of the current macroblock, then predDVIC is set to the differential value of illumination compensation of macroblock “B” and the process is finalized. Otherwise, go to the next step.
(3) If the right-above macroblock “C” of the current macroblock was encoded using macroblock-based illumination change-adaptive motion compensation and its reference index is equal to a reference index of the current macroblock, then predDVIC is set to the differential value of illumination compensation of macroblock “C” and the process is finalized. Otherwise, go to the next step.
(4) If the left-above macroblock “D” of the current macroblock was encoded using macroblock-based illumination change-adaptive motion compensation and its reference index is equal to a reference index of the current macroblock, then predDVIC is set to the differential value of illumination compensation of macroblock “D” and the process is finalized. Otherwise, go to the next step.
(5) If the neighboring macroblocks “A”, “B”, and “C” were encoded using macroblock-based illumination change-adaptive motion compensation, then these three DVICs are median-filtered. predDVIC is set to the result of median-filtering and the process is finalized. Otherwise, go to the next step.
(6) predDVIC is set to zero and the process is finalized.
By using this process, the number of bits for differential value of illumination compensation can be reduced, and then the differential pulse code modulation value of the current differential value of illumination compensation is encoded by the entropy coder. This process shall be performed to reconstruct the differential value of illumination compensation at the decoder.
Since illumination compensation is done on a macroblock basis and only on 16×16 macroblocks, it is observed that it introduces several block artifacts.
The conventional approaches to mitigate coding artifacts were not designed for the specific type of artifact caused by illumination compensation and, thus, cannot fully suppress such artifacts. Since illumination compensation is a coding tool newly introduced in multi-view video coding, no solution have yet been proposed specifically addressing coding artifacts caused by illumination compensation.